Video image processing

ABSTRACT

In a digital video display system, a composite time-compensated video picture, generated from a plurality of input fields, includes discontinuities, e.g. aliases resulting from relative movement of a foreground object and a background. These discontinuities are the results of time-projections from the sequential input fields or frames, which include movement of objects between the respective instants of the frames. These discontinuities are detected and are smoothed by a softening filter, in order to improve the quality of the resulting output image. In a preferred embodiment, the filtering is applied directionally, based on the nature of the discontinuity detected.

FIELD OF THE INVENTION

This invention relates to video image processing and in particular topicture creation with motion compensation image processing.

BACKGROUND

Various aspects of motion compensation are described in our BritishPatent Applications Nos. 9020498.3 and 9111348.0 which describe systemscapable of generating intermediate fields of video image sequences wherethe original sequence of images includes motion of, for example, aforeground object moving over a background. Such intermediate fields arerequired when producing slow motion sequences or for standardsconversion.

In many applications of motion compensation an output image is createdby cutting and pasting one or more parts of various video images onto abackground. The parts of the video images may include motion withinthemselves and the parts as a whole may be moving over the background.

When such cutting and pasting techniques are used visiblediscontinuities are often produced in the output image when there is asharp transition from foreground image to background image. Thesediscontinuities are referred to as `aliases` and the problem of these iswell known in picture creation systems. They are removed from picturesby use of `anti-aliasing` techniques which blend the foreground andbackground pictures together over a few pixels.

SUMMARY OF THE INVENTION

Preferred embodiments of the present invention provide a system whichdetects these discontinuities in ouput images and then blends theforeground and background images together over a few pixels.

BRIEF FIGURE DESCRIPTION

The invention will now be described in detail by way of example withreference to the accompanying drawings in which:

FIG. 1 shows two input fields from an original video sequence with anIntermediate ouput field; and

FIG. 2 shows circuit for removing discontinuities from the output field.

FIG. 3 shows a more complex circuit embodying the invention.

DETAILED DESCRIPTION

A discontinuity does not occur if an area of the picture is simplyshifted, i.e. all of the image was moved by the same amount.

A discontinuity will however occur if the shift between two adjacentelements of a picture is different.

The shift applied to a picture element (pixel) is the motion velocity ofthat pixel multiplied by time between the input and output pictures. Inregions of obscured and revealed background i.e. moving foregroundobject, the time vector is distorted so that parts of the image aretaken from either the input field before or the input field after theoutput field time. This is called the cross-fade signal.

In a two field motion compensation algorithm the output picture is madefrom two sequential input fields and two motion vector fields. The twovector fields are associated with the forward projection of the earlierinput fields and the backward projection of the following picture field.

The first step in identifying a discontinuity in an output field is tocalculate the displacement from the velocity vectors multiplied by thedistance in time. This displacement is termed the shift field and isthen passed to a small two dimensional differentiating filter to lookfor transitions in the shift field. If the output of the filter is abovea predetermined threshold then a discontinuity has been found. Thethreshold is set at a level of say half a pixel. A low pass filter isthen applied to the discontinuity to soften the transition between thetwo pictures. This softening is equivalent to a blending of the twopictures by taking different proportions of each image when generating apixel.

The softening filter only operates on significant fractures in thepicture. Tiny differences between various measurements of the samemotion are invisible in the output picture and the unnecessary use of ablurring filter may do more damage than good.

FIG. 1 shows the situation of a foreground object moving upwards and thebackground moving downwards. Several different situations arise andthese are described below:

1) A-B Background is visible in both input pictures and both inputpictures contribute to the output picture.

2) B-C Background is only visible in the first picture F1, and theoutput picture is taken from F1 only. At point B there is no change inthe forward projection fields but there is in the backward shift fieldsfrom F2. The switch at point B is only visible if camera lag has blurredthe foreground object.

3) C-D Foreground is visible in F1 and F2 and output pictures takeinformation from both. At point C there is a change in both the forwardsshift field and in the backwards shift field. This can give a visiblediscontinuity in the output picture and softening may be required.

4) D-E Background is only visible in the second picture F1, the outputpicture is taken from F2 only. At point D there is a change in both theforwards shift field and in the backwards shift field. This can give avisible discontinuity in the output picture.

5) E-F Background is visible in both pictures and both input picturescontribute to the output picture. At point E there is no change in thebackwards shift fields bun there is in the forwards shift field. Theswitch at point E is only visible if camera lag has blurred theforground object.

The technique used detects the cut and paste edges in the picture andapplies filtering subject to the probability of the edges being visiblei.e. aliases being present.

In picture creation the output picture is made from the weighted sum oftwo input pictures. At an output time close to an input time the pictureis made primarily from the nearest field in time. The nearest field isunlikely to have many positional fractures because the picture elementshave not yet moved far. However, the further field is quite likely tohave fractures but the contribution from this field is less because itis further away in time. In the limit, if the output picture is co-timedwith the input picture then there is no fracture in the output pictureas the input pictures makes 100% of the output picture. There are,however, lots of fractures in the other picture but that makes 0% of theoutput.

Another rule is used to stop the low pass filter being applied toperfectly good output. According to this rule there is a fractureexisting in the output picture if there is a fracture in thedisplacement field belonging to an input picture and the contributionfrom that picture is more than F% (F may be set to about 20% but thiscan be optimised for a particular application).

There are therefore three reasons why there may be a visible fracture inan output picture:

1) There is a fracture from the first displacement field and itscontribution is above F%.

2) There is a fracture from the second displacement field and itscontribution is above F%.

3) There is a change in the cross-fade factor greater than C% at theoutput field.

Since any one may cause a visible fracture the low pass filter isassumed to be necessary. The filter itself is a very small aperture andthe side effect is quite small but quite effective at breaking up thesharp discontinuities caused by cut and paste.

The pictures are built the way described in previous work but inaddition the complete picture is passed to a three by three tap low-passfilter. A switch signal is derived in the way outlined above and is usedto select either sharp unfiltered picture or the softer filteredpicture. The switch signal along a fracture line is normally only a fewpixels wide, say six at the most, so that the area of the pictureaffected by the airbrush filter is a minimum. It is difficult to see theeffect of the filter on picture sharpness because it may affect only twoor three samples.

FIG. 2 shows the arrangement of the low-pass filter and switch. A delayto match any delay introduced by the filter is provided in parallel forunfiltered output.

The system described can be applied to any system which uses motioncompensation.

FIG. 3 shows a more sophisticated embodiment of the invention. Thisembodiment searches for five different types of fracture and performssoftening if one of these is detected. The five types are x and ydirectional fractures in data from a forward projected vector field, xand y fractures in data form a reverse projected vector field. Andfinally fractures in crossfaded areas of the picture. Thus forwardfracture logic, reverse fracture logic, and crossfade fracture logic areshown.

The forward and reverse projected vector fields are derived as follows.Vectors are assigned to each elementary area of the input fields forforward and reverse projection. These vectors are then assigned toelementary areas of an output field in a shifted vector field bymultiplying them by the distance to the output field to give a totaldisplacement which is then added to the base address of the pixelassigned to that vector in the input field. The shifted vector fieldthus comprises a set of unscaled vectors each assigned to an elementaryarea of the output field.

The x and y components of a forward projected vector field are each putthrough 2D differentiating filters 10 and 12 respectively. Thisgenerates a signal, in each case, whose magnitude is dependent upon thedifference between vectors assigned to adjacent elementary areas (orpixels) of the picture.

The output of the differentiating filters is then scaled in multipliers11 and 13 which multiply the differential by the distance of the outputfield from the input field. This is necessary to determine the size ofthe fracture. Any sign changes in the differentials are then removed inthe modulus detectors 15 and 17.

A pair of threshold detectors 14 and 16 receive the outputs of thesemodulus detectors and if the output of either is above a threshold B,i.e. a discontinuity above a predetermined value has been detected, thena control signal is generated at the output of the respective detector.

The control signals at the outputs of the two detectors are passed to anOR gate 18 which generates an output if a signal is produced by eitherdetector.

The output of the OR gate then forms one input to gate 20. The otherinput receives the crossfade factor between the input fields. If F1,which is the proportion of the output field generated by the forwardprojected input field, is greater than 20%, then the AND gate 20 willproduce an output labelled F when a discontinuity is detected. Further2D filters 22 and 24, multipliers 21 and 23, modulus detectors 27 and29, threshold detectors 26 and 28, an OR gate 30, and an AND gate 32operate on the backwards projected vector fields. For the reverseprojected field the outputs of the differentiating filters 22 and 27 aremultiplied by 1 minus distance from the input field. F2, the proportionof the output field generated by the backwards projected vector field is1-CROSSFADE FACTOR. The AND gate 32 has an output labelled R which isgenerated when a discontinuity is detected.

Changes in the crossfade factor due to movement in the picture revealingand obscuring background are detected in a further 2D filter 34. Thisproduces a signal CF from a threshold detector 36 when the threshold isexceeded.

The multipliers may be omitted by varying the thresholds of thethreshold detectors in dependence on the distance of the output fieldfrom the input field and storing a scaled vector in the shifted vectorfield.

The signals F, R, and CF all form inputs to an OR gate 38 and thepresence of any one of them generates an output which controls a switch40 to switch an input video signal through an airbrush filter 40. (seeFIG. 2).

The 2D differentiating filters used are typically 5-tap filters and givea weighting of 4 to the central pixel and weightings of -1 to eachadjacent pixel in the x and y directions.

In a more sophisticated system filters can be used which detect themagnitude of discontinuities in the x and y directions and in responseto these switch in airbrush filters which provide a smoothing effect inthe direction of the discontinuities.

In this the outputs of all the threshold detectors are monitored and asmoothing filter is selected in accordance with the direction of thediscontinuity. If a discontinuity only appears in the x direction thensmoothing will only be performed in the x direction. Correspondingly adiscontinuity in the y direction will be smoothed in the y direction.The smoothing filter used can be selected with reference to a look-uptable. Alternatively the filter direction could be calculated independence on the direction of the fracture detected.

It will be clear to those skilled in the art that although the presentinvention can be implemented in dedicated hardware it can also beimplemented purely in software.

What is claimed is:
 1. A method of improving a video image by detecting,and compensating for, discontinuities in a motion-compensated compositepicture generated from a plurality of input fields, wherein thediscontinuities arise from differential motions between adjacent areasof an input field, comprising the steps of:assigning motion vectors toelementary areas of each of a plurality of input fields in a sequence ofvideo images; generating a shift field from a time projection of themotion vectors assigned to at least one of the plurality of input fieldsin the sequence of video images, detecting transitions in displacementin the shift field between adjacent areas of the shift field,anddisguising, by use of a filter, any transition detected.
 2. A methodaccording to claim 1, in which only transitions above a predeterminedthreshold are detected.
 3. A method according to claim 1 in which thestep of disguising a transition comprises applying a softening filter toareas of the composite picture corresponding to any detected transition.4. A method according to claim 3, in which a softening filter is onlyapplied to a transition detected from an output field which contributesmore than a predetermined proportion of the output field.
 5. A methodaccording to any one of the preceding claims, further comprising thestep of detecting the direction of any discontinuity which is detected.6. A method according to claim 5, in which the softening filter isapplied in a particular direction, which direction is dependent on thedirection of the discontinuity.
 7. A method according to claim 1, inwhich the shift field generated comprises a shifted vector field.
 8. Anapparatus for improving a video image by detecting, and compensatingfor, discontinuities in a motion-compensated composite picture generatedfrom a plurality of input fields wherein the discontinuities arise fromdifferential motion between adjacent areas of an input filedcomprisingmeans for assigning motion vectors to elementary areas of eachinput field in a sequence of video images; means for generating a shiftfield from a time projection of the motion vectors assigned to at leastone of the plurality of input fields, means for detecting transitions inthe vectors of the shift field between adjacent areas of the shift fieldand means for disguising any transition detected by filtering at thattransition, prior to display of a motion-compensated picture.
 9. Anapparatus for detecting discontinuities in a motion-compensatedcomposite picture, generated from a plurality of input fields, accordingto claim 8, in which the detecting means comprisesdifferentiating filtermeans, receiving the shift field, and means for generating a controlsignal when the output of the filter means exceeds a predeterminedlevel.
 10. An apparatus, for detecting discontinuities in amotion-compensated composite picture generated from a plurality of inputfields according to claim 8 or 9, in which the shift field comprises ashifted vector field.
 11. An apparatus according to claim 9 fordetecting discontinuities in a motion-compensated composite picturegenerated from a plurality of input fields, includingmeans responsive tothe control signal to apply a softening filter to an area of thecomposite picture.